Week 04

Oct 24

img = Image.open(test_image[1]) img.save('name.png')

from PIL import Image !zip -r /content/foo.zip /content/valid/

Oct 25

• link to environment features

• link to the canopy coverage outside of the sewer pipe, how it links to root find in the pipe

• link to the type of soil, how it links to infiltration

• Traffic, how it links to defect (fatigue)

• find the average f1 and f2 score excluding ND

• Create distribution of train and validation distribution for each label of each model train

• how can we cache folder in the slurm tempory directory - so that I dont have to zip and unzip my data - ask compute canada.

  • No we can't

• increase the batch size with 9 videos to find the limit.

  • CUDA out of memory when batch size = 64.

• run all 33 videos with batchsize 32

• process to validate the access database

• process to validate the observation in the video

• process to use access database to enhance the observation

• histogram of defect in each video of all 33 videos

• histogram of defect in all 33 videos

• histogram of training and validation dataset

• First time using wandb in compute canada

  • wandb login in the terminal
  • then call the wandb?
  • check inside .netrc if I have the key, do not login to wandb inside your script
  • run the test model to test and load the test model to see if it still depends on the script. (no)

Oct 26

• There is no difference between F2 CIW scores of half size and full size SewerML models. However, F2 score of models trained only on training dataset (14 folders doing 80/20 split) is approximately 0.4 higher than SewerML models trained on (14 training folders and 2 validation folders)
  • compare the performance of SewerML_v1 model with SewerML_v2 model by running prediction on DNV images
  • compare the performance of half size SewerML model with full size image SewerML model by running prediction on DNV images
• There is no significantly difference between F2 CIW scores of model without VA and with VA image label.
  • compare the performance of no_VA SewerML model with VA SewerML model by running prediction on DNV images.
• Run the SewerML model on test image and send the csv file to Joakim.
• Maybe group all the tap break and tap break factory into one category...

Oct 27

Meeting with Sudhir

1. CCTV image from client in Cincinnati can be included in the publication (no ND agreement)
2. There are three major problems with our model:
   1. The image is labelled according to the distance where inspectors input their observations.
      • The problem is the defect might be visible before and after those distances.
      • Look at the video manually to see where within the distance from the annotated distance that defect can be seen.
      • Ask Pavan to create a utility function where the inputs are the current frame, and the number of previous frames you want to do spatial correlation with. Then, plot the spatial correlation vs the number of frames from the selected frame to find the spatial correlation threshold.
   2. Some labels are too specific. For example, from the straight view, the inspector could not tell whether the tap break is active or abandoned. He/She will need to zoom in and move the camera around to determine the state of the tap break.
      • Need to find out how to handle look around situation
   3. There is class imbalance in the dataset
      • Find out a way to handle class imbalance. Can SMOTE handle this?
      • We can try duplicate frames 30 times. Right now, we grab one frame per second. The original video has 30fps. The name of our image is in the interval of 30. The duplicated frame can be labelled as [31, 32,....59]
3. Create a function to get the distance where the label is entered, the label, and the name of the frame that the label is first entered at. This will be needed for the plot_spatial_correlation.
4. Create example frames that got labeled the following way:
   1. 10 images from different videos and labels to show the way the inspector labels the observation (defect)
      1. From my observation, I believe that when the inspector detect the defect, they will move the camera to the point with defect and look around the defect. Then, they will move the camera back to the straight position where the camera is pointing to the center of the pipe (the distance will not decrease when they move the camera back. The distance will either remain the same or increase) before they enter the observation.
      2. There are three types of frames when the observation (defect) is present:
         1. frame where camera is pointing to the center of the pipe before the observation is labelled by the inspector (distance < the labelled distance).
         2. frame where camera is moving around the defect area (the camera is not pointing toward the center of the pipe).
         3. frame where camera is moving back from the defect area and pointing to the center of the pipe (distance == labelled distance >= distance from step 2).
      3. When there are more than one label at the same location, the number of frames from the frame where the observation is input to the frames where the observation is first seen will be varied because each label is input one at the time. For example, frame 1 is the first frame we see tap break root. Frame 10 is the first frame where tap break is entered and Frame 15 is the fist frame where root is entered. If we are going to write a code by looking back number of frame from the first labelled frame, we will need to take this into account. It might be better to do it in term of distance than the number of frame.
   2. Images get labelled if they have the same distance as the labelled frame.
   3. Images get labelled if they have the same distance as and have high correlation (>= 0.9) with the labelled frame.This would be a problem to the frames that captured during the look around. The "look around" frame will have a low correlation with the labelled frame.
   4. Images get labelled if they have different distance but within a certain number of frame ( or distance) away from the labelled frame and have high correlation (>=0.9) with the labelled frame.
5. Create a road map on how I would handle images that got mislabelled.
   1. SewerML can be used to help with initial labelling
      1. SewerML performs well in predicting images with root and tap break.
   2. Implement step 4.2, 4.3, and 4.4. Find the best method out of the three
   3. Write a script to organize the images according to their labels.
   4. Use show_image() to determine whether the images are labelled correctly.
   5. Move the images that are labelled incorrectly to the correct label folder
6. Compare the performance of SewerML model that trained without VA label to the SewerML model that trained with VA label by using them to predict DNV images
   1. Mapping NASSCO defect standard to Danish defect standard and vice versa.
      1. Look at literature review
      2. If there are more than one possibility, do the engineering approximation by finding how often the images of the same label got labelled as the certain label from other standard. Mapping the label to the highest matching label from the other standard.
   2. Calculate the metric

Oct 28

1. Finished creating all_frame CSV files for the rest of the videos (40 videos)
2. There is discrepancy in the way inspector operated the camera
   1. Video in DHL Clements Ave Folder (1 video)
      1. There is discrepancy between label shown in the video and the one in Access Database
      2. At distance = 17.4, the video has four labels, the access database only has three labels
      3. Sometime, the text still is visible on the screen even though the camera has moved out from the defect spot.
         1. Need to edit python notebook to handle this nuance.
      4. At 36.8 m, there seem to be defect but the inspector did not label it in the video and it was not labelled in the access database either.
      5. When the water level is high, the camera will have to pan upward (did not point at the center of the pipe).
      6. The camera is moving at the faster rate than the videos from the previous folder (1 DHL Lynn Valley Oct 6 2021 folder)
3. Right now the images from the 33 videos are stored based on their labels in my local computer
   1. Images get labelled if they have the same distance as the labelled frame.
   2. I will need to check images manually and move it to the correct folder. I want to make sure that I will have the correct labels for all the images from the 33 videos first before running another models
   3. I have not implemented the spatial correction methods because there were discrepancy in the frame number of images in the all frame csv file and the output obtained from the plot_spatial_correlation. (Pavan fixed this issue)
4. Deven has shown me on how to create confusion matrix of the prediction on the validation dataset of the multi-labels models. However, the function does not work for multi-label.
5. Collimator was working fine now. The model was complied and successfully output the result after the meeting with Pavan and you on Thursday. I reported the issue to Brian, but he could not find any problem.